{
 "cells": [
  {
   "cell_type": "markdown",
   "id": "c014a89a-05f5-446a-bc53-dd048c6c4997",
   "metadata": {},
   "source": [
    "## 1. LangChain 中的提示模板\n",
    "\n",
    "提示模板(prompt template)是指生成提示的可重复的方式。它包含一个文本字符串（“模板”），可以接收来自最终用户的一组参数并生成提示。\n",
    "\n",
    "在Langchain 中的通过提示模板类（PromptTemplate）来构建提示的语句，本文将详细介绍如何通过PromptTemplate实现基于特定信息的问题回答。"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "eb420814-0aaf-4244-9c4b-70e087c509d7",
   "metadata": {},
   "source": [
    "## 2. 案例体验\n",
    "🔹 本案例需使用 P100 及以上规格运行，请确保运行规格一致，可按照下图切换规格。\n",
    "\n",
    "![](https://modelarts-labs-bj4-v2.obs.cn-north-4.myhuaweicloud.com/case_zoo/chatglm3/image/1.png)\n",
    "\n",
    "🔹 点击Run in ModelArts，将会进入到ModelArts CodeLab中，这时需要你登录华为云账号，如果没有账号，则需要注册一个，且要进行实名认证，参考[《ModelArts准备工作_简易版》](https://developer.huaweicloud.com/develop/aigallery/article/detail?id=4ce709d6-eb25-4fa4-b214-e2e5d6b7919c) 即可完成账号注册和实名认证。 登录之后，等待片刻，即可进入到CodeLab的运行环境\n",
    "\n",
    "🔹 出现 Out Of Memory ，请检查是否为您的参数配置过高导致，修改参数配置，重启kernel或更换更高规格资源进行规避❗❗❗"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "9a579a34-88f2-4fe3-94f1-88a6597a4a13",
   "metadata": {},
   "source": [
    "### 2.1 下载ChatGLM3模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "500aa400-ffa1-4ad4-b47b-35f503571f43",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "INFO:root:Using MoXing-v2.1.0.5d9c87c8-5d9c87c8\n",
      "INFO:root:Using OBS-Python-SDK-3.20.9.1\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import moxing as mox\n",
    "\n",
    "work_dir = '/home/ma-user/work'\n",
    "obs_path = 'obs://dtse-models/tar-models/chatglm3-6b.tar'\n",
    "ma_path = os.path.join(work_dir, 'chatglm3-6b.tar')\n",
    "mox.file.copy(obs_path, ma_path)\n",
    "\n",
    "mox.file.copy_parallel('obs://modelarts-labs-bj4-v2/course/ModelBox/frpc_linux_amd64', '/home/ma-user/work/frpc_linux_amd64')"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "5e4bf118-b1d9-4b08-97f7-610d546df5cf",
   "metadata": {},
   "source": [
    "进入chatglm3目录，解压模型压缩包"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "36b3ae49-ea02-491a-8f21-980840e3c390",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/work\n",
      "chatglm3-6b/\n",
      "chatglm3-6b/.gitattributes\n",
      "chatglm3-6b/MODEL_LICENSE\n",
      "chatglm3-6b/README.md\n",
      "chatglm3-6b/config.json\n",
      "chatglm3-6b/configuration_chatglm.py\n",
      "chatglm3-6b/modeling_chatglm.py\n",
      "chatglm3-6b/pytorch_model-00001-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00002-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00003-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00004-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00005-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00006-of-00007.bin\n",
      "chatglm3-6b/pytorch_model-00007-of-00007.bin\n",
      "chatglm3-6b/pytorch_model.bin.index.json\n",
      "chatglm3-6b/quantization.py\n",
      "chatglm3-6b/tokenization_chatglm.py\n",
      "chatglm3-6b/tokenizer.model\n",
      "chatglm3-6b/tokenizer_config.json\n"
     ]
    }
   ],
   "source": [
    "os.chdir(work_dir)\n",
    "!pwd\n",
    "!tar -xvf chatglm3-6b.tar"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "2a5bc17f-c539-45d9-b80c-d48ebcecea0b",
   "metadata": {},
   "source": [
    "### 2.2 配置环境\n",
    "\n",
    "本案例依赖Python3.10.10及以上环境，因此我们首先创建虚拟环境："
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "86ab2994-2d9c-4404-a017-c5a3b8a8d671",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "/home/ma-user/anaconda3/lib/python3.7/site-packages/requests/__init__.py:91: RequestsDependencyWarning: urllib3 (1.26.12) or chardet (3.0.4) doesn't match a supported version!\n",
      "  RequestsDependencyWarning)\n",
      "Collecting package metadata (current_repodata.json): done\n",
      "Solving environment: failed with repodata from current_repodata.json, will retry with next repodata source.\n",
      "Collecting package metadata (repodata.json): done\n",
      "Solving environment: done\n",
      "\n",
      "## Package Plan ##\n",
      "\n",
      "  environment location: /home/ma-user/anaconda3/envs/python-3.10.10\n",
      "\n",
      "  added / updated specs:\n",
      "    - python=3.10.10\n",
      "\n",
      "\n",
      "The following NEW packages will be INSTALLED:\n",
      "\n",
      "  _libgcc_mutex      anaconda/pkgs/main/linux-64::_libgcc_mutex-0.1-main\n",
      "  _openmp_mutex      anaconda/pkgs/main/linux-64::_openmp_mutex-5.1-1_gnu\n",
      "  bzip2              anaconda/pkgs/main/linux-64::bzip2-1.0.8-h7b6447c_0\n",
      "  ca-certificates    anaconda/pkgs/main/linux-64::ca-certificates-2023.12.12-h06a4308_0\n",
      "  ld_impl_linux-64   anaconda/pkgs/main/linux-64::ld_impl_linux-64-2.38-h1181459_1\n",
      "  libffi             anaconda/pkgs/main/linux-64::libffi-3.4.4-h6a678d5_0\n",
      "  libgcc-ng          anaconda/pkgs/main/linux-64::libgcc-ng-11.2.0-h1234567_1\n",
      "  libgomp            anaconda/pkgs/main/linux-64::libgomp-11.2.0-h1234567_1\n",
      "  libstdcxx-ng       anaconda/pkgs/main/linux-64::libstdcxx-ng-11.2.0-h1234567_1\n",
      "  libuuid            anaconda/pkgs/main/linux-64::libuuid-1.41.5-h5eee18b_0\n",
      "  ncurses            anaconda/pkgs/main/linux-64::ncurses-6.4-h6a678d5_0\n",
      "  openssl            anaconda/pkgs/main/linux-64::openssl-1.1.1w-h7f8727e_0\n",
      "  pip                anaconda/pkgs/main/linux-64::pip-23.3.1-py310h06a4308_0\n",
      "  python             anaconda/pkgs/main/linux-64::python-3.10.10-h7a1cb2a_2\n",
      "  readline           anaconda/pkgs/main/linux-64::readline-8.2-h5eee18b_0\n",
      "  setuptools         anaconda/pkgs/main/linux-64::setuptools-68.2.2-py310h06a4308_0\n",
      "  sqlite             anaconda/pkgs/main/linux-64::sqlite-3.41.2-h5eee18b_0\n",
      "  tk                 anaconda/pkgs/main/linux-64::tk-8.6.12-h1ccaba5_0\n",
      "  tzdata             anaconda/pkgs/main/noarch::tzdata-2023c-h04d1e81_0\n",
      "  wheel              anaconda/pkgs/main/linux-64::wheel-0.41.2-py310h06a4308_0\n",
      "  xz                 anaconda/pkgs/main/linux-64::xz-5.4.5-h5eee18b_0\n",
      "  zlib               anaconda/pkgs/main/linux-64::zlib-1.2.13-h5eee18b_0\n",
      "\n",
      "\n",
      "Preparing transaction: done\n",
      "Verifying transaction: done\n",
      "Executing transaction: done\n",
      "#\n",
      "# To activate this environment, use\n",
      "#\n",
      "#     $ conda activate python-3.10.10\n",
      "#\n",
      "# To deactivate an active environment, use\n",
      "#\n",
      "#     $ conda deactivate\n",
      "\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting ipykernel\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/5d/4b/ffb537e392e730c9a5b02758f9c87077d9087bcb0d957853e13f121e5ea7/ipykernel-6.23.1-py3-none-any.whl (152 kB)\n",
      "Collecting comm>=0.1.1 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/74/f3/b88d7e1dadf741550c56b70d7ce62673354fddb68e143d193ceb80224208/comm-0.1.3-py3-none-any.whl (6.6 kB)\n",
      "Collecting debugpy>=1.6.5 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/1f/19/345c21f6b62acf556c39e4358a22b0ad868fecb462c1041c13513d229b33/debugpy-1.6.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
      "Collecting ipython>=7.23.1 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/09/db/d641ee07f319002393524b6c5a8b47370520dcb2b6166a0972cfe9398c60/ipython-8.10.0-py3-none-any.whl (784 kB)\n",
      "Collecting jupyter-client>=6.1.12 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/07/37/4019d2c41ca333c08dfdfeb84c0fc0368c8defbbd3c8f0c9a530851e5813/jupyter_client-8.2.0-py3-none-any.whl (103 kB)\n",
      "Collecting jupyter-core!=5.0.*,>=4.12 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/41/1e/92a67f333b9335f04ce409799c030dcfb291712658b9d9d13997f7c91e5a/jupyter_core-5.3.0-py3-none-any.whl (93 kB)\n",
      "Collecting matplotlib-inline>=0.1 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/a6/2d/2230afd570c70074e80fd06857ba2bdc5f10c055bd9125665fe276fadb67/matplotlib_inline-0.1.3-py3-none-any.whl (8.2 kB)\n",
      "Collecting nest-asyncio (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/06/e0/93453ebab12f5ce9a9ceda2ff71648b30e5f2ce5bba19ee3c95cbd0aaa67/nest_asyncio-1.5.4-py3-none-any.whl (5.1 kB)\n",
      "Collecting packaging (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/ec/1a/610693ac4ee14fcdf2d9bf3c493370e4f2ef7ae2e19217d7a237ff42367d/packaging-23.2-py3-none-any.whl (53 kB)\n",
      "Collecting psutil (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/6e/c8/784968329c1c67c28cce91991ef9af8a8913aa5a3399a6a8954b1380572f/psutil-5.9.4-cp36-abi3-manylinux_2_12_x86_64.manylinux2010_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (280 kB)\n",
      "Collecting pyzmq>=20 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/79/f4/c33ff6e3d7bfbceecbb2176f75328c897365a519f507d226e44eea74d6d2/pyzmq-25.1.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.1 MB)\n",
      "Collecting tornado>=6.1 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/66/a5/e6da56c03ff61200d5a43cfb75ab09316fc0836aa7ee26b4e9dcbfc3ae85/tornado-6.3.3-cp38-abi3-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (427 kB)\n",
      "Collecting traitlets>=5.4.0 (from ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/77/75/c28e9ef7abec2b7e9ff35aea3e0be6c1aceaf7873c26c95ae1f0d594de71/traitlets-5.9.0-py3-none-any.whl (117 kB)\n",
      "Collecting backcall (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/4c/1c/ff6546b6c12603d8dd1070aa3c3d273ad4c07f5771689a7b69a550e8c951/backcall-0.2.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting decorator (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/d5/50/83c593b07763e1161326b3b8c6686f0f4b0f24d5526546bee538c89837d6/decorator-5.1.1-py3-none-any.whl (9.1 kB)\n",
      "Collecting jedi>=0.16 (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/6d/60/4acda63286ef6023515eb914543ba36496b8929cb7af49ecce63afde09c6/jedi-0.18.2-py2.py3-none-any.whl (1.6 MB)\n",
      "Collecting pickleshare (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/9a/41/220f49aaea88bc6fa6cba8d05ecf24676326156c23b991e80b3f2fc24c77/pickleshare-0.7.5-py2.py3-none-any.whl (6.9 kB)\n",
      "Collecting prompt-toolkit<3.1.0,>=3.0.30 (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/87/3f/1f5a0ff475ae6481f4b0d45d4d911824d3218b94ee2a97a8cb84e5569836/prompt_toolkit-3.0.38-py3-none-any.whl (385 kB)\n",
      "Collecting pygments>=2.4.0 (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/0b/42/d9d95cc461f098f204cd20c85642ae40fbff81f74c300341b8d0e0df14e0/Pygments-2.14.0-py3-none-any.whl (1.1 MB)\n",
      "Collecting stack-data (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/6a/81/aa96c25c27f78cdc444fec27d80f4c05194c591465e491a1358d8a035bc1/stack_data-0.6.2-py3-none-any.whl (24 kB)\n",
      "Collecting pexpect>4.3 (from ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/39/7b/88dbb785881c28a102619d46423cb853b46dbccc70d3ac362d99773a78ce/pexpect-4.8.0-py2.py3-none-any.whl (59 kB)\n",
      "Collecting python-dateutil>=2.8.2 (from jupyter-client>=6.1.12->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/36/7a/87837f39d0296e723bb9b62bbb257d0355c7f6128853c78955f57342a56d/python_dateutil-2.8.2-py2.py3-none-any.whl (247 kB)\n",
      "Collecting platformdirs>=2.5 (from jupyter-core!=5.0.*,>=4.12->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/ba/24/a83a900a90105f8ad3f20df5bb5a2cde886df7125c7827e196e4ed4fa8a7/platformdirs-3.0.0-py3-none-any.whl (14 kB)\n",
      "Collecting parso<0.9.0,>=0.8.0 (from jedi>=0.16->ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/05/63/8011bd08a4111858f79d2b09aad86638490d62fbf881c44e434a6dfca87b/parso-0.8.3-py2.py3-none-any.whl (100 kB)\n",
      "Collecting ptyprocess>=0.5 (from pexpect>4.3->ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/22/a6/858897256d0deac81a172289110f31629fc4cee19b6f01283303e18c8db3/ptyprocess-0.7.0-py2.py3-none-any.whl (13 kB)\n",
      "Collecting wcwidth (from prompt-toolkit<3.1.0,>=3.0.30->ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/59/7c/e39aca596badaf1b78e8f547c807b04dae603a433d3e7a7e04d67f2ef3e5/wcwidth-0.2.5-py2.py3-none-any.whl (30 kB)\n",
      "Collecting six>=1.5 (from python-dateutil>=2.8.2->jupyter-client>=6.1.12->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/d9/5a/e7c31adbe875f2abbb91bd84cf2dc52d792b5a01506781dbcf25c91daf11/six-1.16.0-py2.py3-none-any.whl (11 kB)\n",
      "Collecting executing>=1.2.0 (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/28/3c/bc3819dd8b1a1588c9215a87271b6178cc5498acaa83885211f5d4d9e693/executing-1.2.0-py2.py3-none-any.whl (24 kB)\n",
      "Collecting asttokens>=2.1.0 (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/f3/e1/64679d9d0759db5b182222c81ff322c2fe2c31e156a59afd6e9208c960e5/asttokens-2.2.1-py2.py3-none-any.whl (26 kB)\n",
      "Collecting pure-eval (from stack-data->ipython>=7.23.1->ipykernel)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/2b/27/77f9d5684e6bce929f5cfe18d6cfbe5133013c06cb2fbf5933670e60761d/pure_eval-0.2.2-py3-none-any.whl (11 kB)\n",
      "Installing collected packages: wcwidth, pure-eval, ptyprocess, pickleshare, executing, backcall, traitlets, tornado, six, pyzmq, pygments, psutil, prompt-toolkit, platformdirs, pexpect, parso, packaging, nest-asyncio, decorator, debugpy, python-dateutil, matplotlib-inline, jupyter-core, jedi, comm, asttokens, stack-data, jupyter-client, ipython, ipykernel\n",
      "Successfully installed asttokens-2.2.1 backcall-0.2.0 comm-0.1.3 debugpy-1.6.6 decorator-5.1.1 executing-1.2.0 ipykernel-6.23.1 ipython-8.10.0 jedi-0.18.2 jupyter-client-8.2.0 jupyter-core-5.3.0 matplotlib-inline-0.1.3 nest-asyncio-1.5.4 packaging-23.2 parso-0.8.3 pexpect-4.8.0 pickleshare-0.7.5 platformdirs-3.0.0 prompt-toolkit-3.0.38 psutil-5.9.4 ptyprocess-0.7.0 pure-eval-0.2.2 pygments-2.14.0 python-dateutil-2.8.2 pyzmq-25.1.1 six-1.16.0 stack-data-0.6.2 tornado-6.3.3 traitlets-5.9.0 wcwidth-0.2.5\n"
     ]
    }
   ],
   "source": [
    "!/home/ma-user/anaconda3/bin/conda create -n python-3.10.10 python=3.10.10 -y --override-channels --channel https://mirrors.tuna.tsinghua.edu.cn/anaconda/pkgs/main\n",
    "!/home/ma-user/anaconda3/envs/python-3.10.10/bin/pip install ipykernel"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "738f430c-401d-48d2-b330-2eca3a2a629c",
   "metadata": {},
   "outputs": [],
   "source": [
    "import json\n",
    "import os\n",
    "\n",
    "data = {\n",
    "   \"display_name\": \"python-3.10.10\",\n",
    "   \"env\": {\n",
    "      \"PATH\": \"/home/ma-user/anaconda3/envs/python-3.10.10/bin:/home/ma-user/anaconda3/envs/python-3.7.10/bin:/modelarts/authoring/notebook-conda/bin:/opt/conda/bin:/usr/local/nvidia/bin:/usr/local/cuda/bin:/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/modelarts/ma-cli/bin:/home/ma-user/anaconda3/envs/PyTorch-1.8/bin\"\n",
    "   },\n",
    "   \"language\": \"python\",\n",
    "   \"argv\": [\n",
    "      \"/home/ma-user/anaconda3/envs/python-3.10.10/bin/python\",\n",
    "      \"-m\",\n",
    "      \"ipykernel\",\n",
    "      \"-f\",\n",
    "      \"{connection_file}\"\n",
    "   ]\n",
    "}\n",
    "\n",
    "if not os.path.exists(\"/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/\"):\n",
    "    os.mkdir(\"/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/\")\n",
    "\n",
    "with open('/home/ma-user/anaconda3/share/jupyter/kernels/python-3.10.10/kernel.json', 'w') as f:\n",
    "    json.dump(data, f, indent=4)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "39d8b96b-98bc-489f-becc-998c27b994dc",
   "metadata": {},
   "source": [
    "创建完成后，稍等片刻，或刷新页面，点击右上角kernel选择python-3.10.10\n",
    "\n",
    "![](https://modelarts-labs-bj4-v2.obs.cn-north-4.myhuaweicloud.com/case_zoo/chatglm3/image/2.png)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "283d52cf-12c6-4330-ab15-40f3c0f2493e",
   "metadata": {},
   "source": [
    "### 2.3 安装依赖库"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "6840d348-a071-42d0-b96f-4e6ecff9547d",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting transformers==4.30.2\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/5b/0b/e45d26ccd28568013523e04f325432ea88a442b4e3020b757cf4361f0120/transformers-4.30.2-py3-none-any.whl (7.2 MB)\n",
      "Collecting filelock (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/00/45/ec3407adf6f6b5bf867a4462b2b0af27597a26bd3cd6e2534cb6ab029938/filelock-3.12.2-py3-none-any.whl (10 kB)\n",
      "Collecting huggingface-hub<1.0,>=0.14.1 (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/7f/c4/adcbe9a696c135578cabcbdd7331332daad4d49b7c43688bc2d36b3a47d2/huggingface_hub-0.16.4-py3-none-any.whl (268 kB)\n",
      "Collecting numpy>=1.17 (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/71/3c/3b1981c6a1986adc9ee7db760c0c34ea5b14ac3da9ecfcf1ea2a4ec6c398/numpy-1.25.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (18.2 MB)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (23.2)\n",
      "Collecting pyyaml>=5.1 (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/29/61/bf33c6c85c55bc45a29eee3195848ff2d518d84735eb0e2d8cb42e0d285e/PyYAML-6.0.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (705 kB)\n",
      "Collecting regex!=2019.12.17 (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/7c/81/b064cc2c67ca2182137641f9d3fd47fe470f1a84674d9b9f91fd39bf0e6f/regex-2023.5.5-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (769 kB)\n",
      "Collecting requests (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/70/8e/0e2d847013cb52cd35b38c009bb167a1a26b2ce6cd6965bf26b47bc0bf44/requests-2.31.0-py3-none-any.whl (62 kB)\n",
      "Collecting tokenizers!=0.11.3,<0.14,>=0.11.1 (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/94/60/ff26cce378023624ffcad91edaa4871f561d6ba7295185c45037ddba80e2/tokenizers-0.13.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (7.8 MB)\n",
      "Collecting safetensors>=0.3.1 (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/6c/f0/c17bbdb1e5f9dab29d44cade445135789f75f8f08ea2728d04493ea8412b/safetensors-0.3.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "Collecting tqdm>=4.27 (from transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/00/e5/f12a80907d0884e6dff9c16d0c0114d81b8cd07dc3ae54c5e962cc83037e/tqdm-4.66.1-py3-none-any.whl (78 kB)\n",
      "Collecting fsspec (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/37/57/eb7c3c10b187d3b8565946772ce0229c79e3c623010eda0aeb5032ff56f4/fsspec-2022.11.0-py3-none-any.whl (139 kB)\n",
      "Collecting typing-extensions>=3.7.4.3 (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/24/21/7d397a4b7934ff4028987914ac1044d3b7d52712f30e2ac7a2ae5bc86dd0/typing_extensions-4.8.0-py3-none-any.whl (31 kB)\n",
      "Collecting charset-normalizer<4,>=2 (from requests->transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/da/f1/3702ba2a7470666a62fd81c58a4c40be00670e5006a67f4d626e57f013ae/charset_normalizer-3.3.2-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (142 kB)\n",
      "Collecting idna<4,>=2.5 (from requests->transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/fc/34/3030de6f1370931b9dbb4dad48f6ab1015ab1d32447850b9fc94e60097be/idna-3.4-py3-none-any.whl (61 kB)\n",
      "Collecting urllib3<3,>=1.21.1 (from requests->transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/96/94/c31f58c7a7f470d5665935262ebd7455c7e4c7782eb525658d3dbf4b9403/urllib3-2.1.0-py3-none-any.whl (104 kB)\n",
      "Collecting certifi>=2017.4.17 (from requests->transformers==4.30.2)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/64/62/428ef076be88fa93716b576e4a01f919d25968913e817077a386fcbe4f42/certifi-2023.11.17-py3-none-any.whl (162 kB)\n",
      "Installing collected packages: tokenizers, safetensors, urllib3, typing-extensions, tqdm, regex, pyyaml, numpy, idna, fsspec, filelock, charset-normalizer, certifi, requests, huggingface-hub, transformers\n",
      "Successfully installed certifi-2023.11.17 charset-normalizer-3.3.2 filelock-3.12.2 fsspec-2022.11.0 huggingface-hub-0.16.4 idna-3.4 numpy-1.25.2 pyyaml-6.0.1 regex-2023.5.5 requests-2.31.0 safetensors-0.3.3 tokenizers-0.13.3 tqdm-4.66.1 transformers-4.30.2 typing-extensions-4.8.0 urllib3-2.1.0\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting sentencepiece==0.1.99\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/7f/e5/323dc813b3e1339305f888d035e2f3725084fc4dcf051995b366dd26cc90/sentencepiece-0.1.99-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.3 MB)\n",
      "Installing collected packages: sentencepiece\n",
      "Successfully installed sentencepiece-0.1.99\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Collecting torch==2.0.1\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/8c/4d/17e07377c9c3d1a0c4eb3fde1c7c16b5a0ce6133ddbabc08ceef6b7f2645/torch-2.0.1-cp310-cp310-manylinux1_x86_64.whl (619.9 MB)\n",
      "Requirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (3.12.2)\n",
      "Requirement already satisfied: typing-extensions in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (4.8.0)\n",
      "Collecting sympy (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/d2/05/e6600db80270777c4a64238a98d442f0fd07cc8915be2a1c16da7f2b9e74/sympy-1.12-py3-none-any.whl (5.7 MB)\n",
      "Collecting networkx (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/42/31/d2f89f1ae42718f8c8a9e440ebe38d7d5fe1e0d9eb9178ce779e365b3ab0/networkx-2.8.8-py3-none-any.whl (2.0 MB)\n",
      "Collecting jinja2 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/bc/c3/f068337a370801f372f2f8f6bad74a5c140f6fda3d9de154052708dd3c65/Jinja2-3.1.2-py3-none-any.whl (133 kB)\n",
      "Collecting nvidia-cuda-nvrtc-cu11==11.7.99 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/ea/8d/0709ba16c2831c17ec1c2ea1eeb89ada11ffa8d966d773cce0a7463b22bb/nvidia_cuda_nvrtc_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (21.0 MB)\n",
      "Collecting nvidia-cuda-runtime-cu11==11.7.99 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/36/92/89cf558b514125d2ebd8344dd2f0533404b416486ff681d5434a5832a019/nvidia_cuda_runtime_cu11-11.7.99-py3-none-manylinux1_x86_64.whl (849 kB)\n",
      "Collecting nvidia-cuda-cupti-cu11==11.7.101 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/e6/9d/dd0cdcd800e642e3c82ee3b5987c751afd4f3fb9cc2752517f42c3bc6e49/nvidia_cuda_cupti_cu11-11.7.101-py3-none-manylinux1_x86_64.whl (11.8 MB)\n",
      "Collecting nvidia-cudnn-cu11==8.5.0.96 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/dc/30/66d4347d6e864334da5bb1c7571305e501dcb11b9155971421bb7bb5315f/nvidia_cudnn_cu11-8.5.0.96-2-py3-none-manylinux1_x86_64.whl (557.1 MB)\n",
      "Collecting nvidia-cublas-cu11==11.10.3.66 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/ce/41/fdeb62b5437996e841d83d7d2714ca75b886547ee8017ee2fe6ea409d983/nvidia_cublas_cu11-11.10.3.66-py3-none-manylinux1_x86_64.whl (317.1 MB)\n",
      "Collecting nvidia-cufft-cu11==10.9.0.58 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/74/79/b912a77e38e41f15a0581a59f5c3548d1ddfdda3225936fb67c342719e7a/nvidia_cufft_cu11-10.9.0.58-py3-none-manylinux1_x86_64.whl (168.4 MB)\n",
      "Collecting nvidia-curand-cu11==10.2.10.91 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/8f/11/af78d54b2420e64a4dd19e704f5bb69dcb5a6a3138b4465d6a48cdf59a21/nvidia_curand_cu11-10.2.10.91-py3-none-manylinux1_x86_64.whl (54.6 MB)\n",
      "Collecting nvidia-cusolver-cu11==11.4.0.1 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/25/4b/272f9aa7838e545b47878e4aec4f09b0fecf17dbd312cf5c5dc398b0637f/nvidia_cusolver_cu11-11.4.0.1-py3-none-manylinux1_x86_64.whl (102.6 MB)\n",
      "Collecting nvidia-cusparse-cu11==11.7.4.91 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/ea/6f/6d032cc1bb7db88a989ddce3f4968419a7edeafda362847f42f614b1f845/nvidia_cusparse_cu11-11.7.4.91-py3-none-manylinux1_x86_64.whl (173.2 MB)\n",
      "Collecting nvidia-nccl-cu11==2.14.3 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/55/92/914cdb650b6a5d1478f83148597a25e90ea37d739bd563c5096b0e8a5f43/nvidia_nccl_cu11-2.14.3-py3-none-manylinux1_x86_64.whl (177.1 MB)\n",
      "Collecting nvidia-nvtx-cu11==11.7.91 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/23/d5/09493ff0e64fd77523afbbb075108f27a13790479efe86b9ffb4587671b5/nvidia_nvtx_cu11-11.7.91-py3-none-manylinux1_x86_64.whl (98 kB)\n",
      "Collecting triton==2.0.0 (from torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/ca/31/ff6be541195daf77aa5c72303b2354661a69e717967d44d91eb4f3fdce32/triton-2.0.0-1-cp310-cp310-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (63.3 MB)\n",
      "Requirement already satisfied: setuptools in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (68.2.2)\n",
      "Requirement already satisfied: wheel in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (0.41.2)\n",
      "Collecting cmake (from triton==2.0.0->torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/2e/51/3a4672a819b4532a378bfefad8f886cfe71057556e0d4eefb64523fd370a/cmake-3.27.2-py2.py3-none-manylinux2014_x86_64.manylinux_2_17_x86_64.whl (26.1 MB)\n",
      "Collecting lit (from triton==2.0.0->torch==2.0.1)\n",
      "  Using cached lit-16.0.5-py3-none-any.whl\n",
      "Collecting MarkupSafe>=2.0 (from jinja2->torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/12/b3/d9ed2c0971e1435b8a62354b18d3060b66c8cb1d368399ec0b9baa7c0ee5/MarkupSafe-2.1.3-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (25 kB)\n",
      "Collecting mpmath>=0.19 (from sympy->torch==2.0.1)\n",
      "  Using cached http://repo.myhuaweicloud.com/repository/pypi/packages/43/e3/7d92a15f894aa0c9c4b49b8ee9ac9850d6e63b03c9c32c0367a13ae62209/mpmath-1.3.0-py3-none-any.whl (536 kB)\n",
      "Installing collected packages: mpmath, lit, cmake, sympy, nvidia-nvtx-cu11, nvidia-nccl-cu11, nvidia-cusparse-cu11, nvidia-cusolver-cu11, nvidia-curand-cu11, nvidia-cufft-cu11, nvidia-cuda-runtime-cu11, nvidia-cuda-nvrtc-cu11, nvidia-cuda-cupti-cu11, nvidia-cublas-cu11, networkx, MarkupSafe, nvidia-cudnn-cu11, jinja2, triton, torch\n",
      "Successfully installed MarkupSafe-2.1.3 cmake-3.27.2 jinja2-3.1.2 lit-16.0.5 mpmath-1.3.0 networkx-2.8.8 nvidia-cublas-cu11-11.10.3.66 nvidia-cuda-cupti-cu11-11.7.101 nvidia-cuda-nvrtc-cu11-11.7.99 nvidia-cuda-runtime-cu11-11.7.99 nvidia-cudnn-cu11-8.5.0.96 nvidia-cufft-cu11-10.9.0.58 nvidia-curand-cu11-10.2.10.91 nvidia-cusolver-cu11-11.4.0.1 nvidia-cusparse-cu11-11.7.4.91 nvidia-nccl-cu11-2.14.3 nvidia-nvtx-cu11-11.7.91 sympy-1.12 torch-2.0.1 triton-2.0.0\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Collecting langchain==0.0.329\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/42/4e/86204994aeb2e4ac367a7fade896b13532eae2430299052eb2c80ca35d2c/langchain-0.0.329-py3-none-any.whl (2.0 MB)\n",
      "Requirement already satisfied: PyYAML>=5.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (6.0.1)\n",
      "Collecting SQLAlchemy<3,>=1.4 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/aa/1c/0b66318368b1c9ef51c5c8560530b8ef842164e10eea08cacb06739388e0/SQLAlchemy-2.0.23-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (3.0 MB)\n",
      "Collecting aiohttp<4.0.0,>=3.8.3 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/2f/16/50441c4baa39e5426181c6f630203ab65029f9a9c55d0a1019a31c26d702/aiohttp-3.9.1-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (1.2 MB)\n",
      "Collecting anyio<4.0 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/19/24/44299477fe7dcc9cb58d0a57d5a7588d6af2ff403fdd2d47a246c91a3246/anyio-3.7.1-py3-none-any.whl (80 kB)\n",
      "Collecting async-timeout<5.0.0,>=4.0.0 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/a7/fa/e01228c2938de91d47b307831c62ab9e4001e747789d0b05baf779a6488c/async_timeout-4.0.3-py3-none-any.whl (5.7 kB)\n",
      "Collecting dataclasses-json<0.7,>=0.5.7 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ae/53/8c006de775834cd4ea64a445402dc195caeebb77dc76b7defb9b3887cb0d/dataclasses_json-0.6.3-py3-none-any.whl (28 kB)\n",
      "Collecting jsonpatch<2.0,>=1.33 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/73/07/02e16ed01e04a374e644b575638ec7987ae846d25ad97bcc9945a3ee4b0e/jsonpatch-1.33-py2.py3-none-any.whl (12 kB)\n",
      "Collecting langsmith<0.1.0,>=0.0.52 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c0/a2/7814b2341d2919f8305cdaff2e37f76b04c45839f402a38cf13ef7153bea/langsmith-0.0.75-py3-none-any.whl (46 kB)\n",
      "Requirement already satisfied: numpy<2,>=1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (1.25.2)\n",
      "Collecting pydantic<3,>=1 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/dd/b7/9aea7ee6c01fe3f3c03b8ca3c7797c866df5fecece9d6cb27caa138db2e2/pydantic-2.5.3-py3-none-any.whl (381 kB)\n",
      "Requirement already satisfied: requests<3,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from langchain==0.0.329) (2.31.0)\n",
      "Collecting tenacity<9.0.0,>=8.1.0 (from langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f4/f1/990741d5bb2487d529d20a433210ffa136a367751e454214013b441c4575/tenacity-8.2.3-py3-none-any.whl (24 kB)\n",
      "Collecting attrs>=17.3.0 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/f0/eb/fcb708c7bf5056045e9e98f62b93bd7467eb718b0202e7698eb11d66416c/attrs-23.1.0-py3-none-any.whl (61 kB)\n",
      "Collecting multidict<7.0,>=4.5 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/56/b5/ac112889bfc68e6cf4eda1e4325789b166c51c6cd29d5633e28fb2c2f966/multidict-6.0.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (114 kB)\n",
      "Collecting yarl<2.0,>=1.0 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/0ade1409d184cbc9e85acd403a386a7c0563b92ff0f26d138ff9e86e48b4/yarl-1.9.4-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (301 kB)\n",
      "Collecting frozenlist>=1.1.1 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ec/25/0c87df2e53c0c5d90f7517ca0ff7aca78d050a8ec4d32c4278e8c0e52e51/frozenlist-1.4.1-cp310-cp310-manylinux_2_5_x86_64.manylinux1_x86_64.manylinux_2_17_x86_64.manylinux2014_x86_64.whl (239 kB)\n",
      "Collecting aiosignal>=1.1.2 (from aiohttp<4.0.0,>=3.8.3->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/76/ac/a7305707cb852b7e16ff80eaf5692309bde30e2b1100a1fcacdc8f731d97/aiosignal-1.3.1-py3-none-any.whl (7.6 kB)\n",
      "Requirement already satisfied: idna>=2.8 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from anyio<4.0->langchain==0.0.329) (3.4)\n",
      "Collecting sniffio>=1.1 (from anyio<4.0->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/c3/a0/5dba8ed157b0136607c7f2151db695885606968d1fae123dc3391e0cfdbf/sniffio-1.3.0-py3-none-any.whl (10 kB)\n",
      "Collecting exceptiongroup (from anyio<4.0->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/b8/9a/5028fd52db10e600f1c4674441b968cf2ea4959085bfb5b99fb1250e5f68/exceptiongroup-1.2.0-py3-none-any.whl (16 kB)\n",
      "Collecting marshmallow<4.0.0,>=3.18.0 (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/ed/3c/cebfdcad015240014ff08b883d1c0c427f2ba45ae8c6572851b6ef136cad/marshmallow-3.20.1-py3-none-any.whl (49 kB)\n",
      "Collecting typing-inspect<1,>=0.4.0 (from dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/65/f3/107a22063bf27bdccf2024833d3445f4eea42b2e598abfbd46f6a63b6cb0/typing_inspect-0.9.0-py3-none-any.whl (8.8 kB)\n",
      "Collecting jsonpointer>=1.9 (from jsonpatch<2.0,>=1.33->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/12/f6/0232cc0c617e195f06f810534d00b74d2f348fe71b2118009ad8ad31f878/jsonpointer-2.4-py2.py3-none-any.whl (7.8 kB)\n",
      "Collecting annotated-types>=0.4.0 (from pydantic<3,>=1->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/28/78/d31230046e58c207284c6b2c4e8d96e6d3cb4e52354721b944d3e1ee4aa5/annotated_types-0.6.0-py3-none-any.whl (12 kB)\n",
      "Collecting pydantic-core==2.14.6 (from pydantic<3,>=1->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/90/28/3c6843e6b203999be2660d3f114be196f2182dcac533dc764ad320c9184d/pydantic_core-2.14.6-cp310-cp310-manylinux_2_17_x86_64.manylinux2014_x86_64.whl (2.1 MB)\n",
      "Requirement already satisfied: typing-extensions>=4.6.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pydantic<3,>=1->langchain==0.0.329) (4.8.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (3.3.2)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests<3,>=2->langchain==0.0.329) (2023.11.17)\n",
      "Collecting greenlet!=0.4.17 (from SQLAlchemy<3,>=1.4->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/24/35/945d5b10648fec9b20bcc6df8952d20bb3bba76413cd71c1fdbee98f5616/greenlet-3.0.3-cp310-cp310-manylinux_2_24_x86_64.manylinux_2_28_x86_64.whl (616 kB)\n",
      "Requirement already satisfied: packaging>=17.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from marshmallow<4.0.0,>=3.18.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.329) (23.2)\n",
      "Collecting mypy-extensions>=0.3.0 (from typing-inspect<1,>=0.4.0->dataclasses-json<0.7,>=0.5.7->langchain==0.0.329)\n",
      "  Using cached https://pypi.tuna.tsinghua.edu.cn/packages/2a/e2/5d3f6ada4297caebe1a2add3b126fe800c96f56dbe5d1988a2cbe0b267aa/mypy_extensions-1.0.0-py3-none-any.whl (4.7 kB)\n",
      "Installing collected packages: tenacity, sniffio, pydantic-core, mypy-extensions, multidict, marshmallow, jsonpointer, greenlet, frozenlist, exceptiongroup, attrs, async-timeout, annotated-types, yarl, typing-inspect, SQLAlchemy, pydantic, jsonpatch, anyio, aiosignal, langsmith, dataclasses-json, aiohttp, langchain\n",
      "Successfully installed SQLAlchemy-2.0.23 aiohttp-3.9.1 aiosignal-1.3.1 annotated-types-0.6.0 anyio-3.7.1 async-timeout-4.0.3 attrs-23.1.0 dataclasses-json-0.6.3 exceptiongroup-1.2.0 frozenlist-1.4.1 greenlet-3.0.3 jsonpatch-1.33 jsonpointer-2.4 langchain-0.0.329 langsmith-0.0.75 marshmallow-3.20.1 multidict-6.0.4 mypy-extensions-1.0.0 pydantic-2.5.3 pydantic-core-2.14.6 sniffio-1.3.0 tenacity-8.2.3 typing-inspect-0.9.0 yarl-1.9.4\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers==4.30.2\n",
    "!pip install sentencepiece==0.1.99\n",
    "!pip install torch==2.0.1\n",
    "!pip install langchain==0.0.329 -i https://pypi.tuna.tsinghua.edu.cn/simple"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "f6703d3e-8451-47e7-bac8-401cdb039be7",
   "metadata": {},
   "source": [
    "## 3. 代码开发\n",
    "\n",
    "首先加载ChatGLM3模型，然后基于langchain的PromptTemplate类来实现提示模板定义，具体代码如下：\n",
    "\n",
    "### 3.1 基于langchain加载ChatGLM3模型"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 2,
   "id": "7e589553-2617-4a2c-a330-d4d6029d21da",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "/home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/tqdm/auto.py:21: TqdmWarning: IProgress not found. Please update jupyter and ipywidgets. See https://ipywidgets.readthedocs.io/en/stable/user_install.html\n",
      "  from .autonotebook import tqdm as notebook_tqdm\n",
      "Loading checkpoint shards: 100%|██████████| 7/7 [00:07<00:00,  1.10s/it]\n"
     ]
    }
   ],
   "source": [
    "import os\n",
    "from typing import Dict, Union, Optional\n",
    "from typing import List\n",
    "\n",
    "from langchain.llms.base import LLM\n",
    "from langchain.llms.utils import enforce_stop_tokens\n",
    "from transformers import AutoModel, AutoTokenizer\n",
    "\n",
    "#新建ChatGLMService类\n",
    "class ChatGLMService(LLM):\n",
    "    max_token: int = 10000\n",
    "    temperature: float = 0.1\n",
    "    top_p = 0.9\n",
    "    history = []\n",
    "    tokenizer: object = None\n",
    "    model: object = None\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    #定义llm模型类型\n",
    "    @property\n",
    "    def _llm_type(self) -> str:\n",
    "        return \"ChatGLM\"\n",
    "\n",
    "    #定义类调用的逻辑\n",
    "    def _call(self,\n",
    "              prompt: str,\n",
    "              stop: Optional[List[str]] = None) -> str:\n",
    "        response, history = self.model.chat(\n",
    "            self.tokenizer,\n",
    "            prompt,\n",
    "            history=self.history,\n",
    "            max_length=self.max_token,\n",
    "            temperature=self.temperature,\n",
    "        )\n",
    "        if stop is not None:\n",
    "            response = enforce_stop_tokens(response, stop)#基于结束词将内容切断\n",
    "            \n",
    "        self.history = self.history + [history[-1]]#将问题和答案传到历史记录中\n",
    "        return response\n",
    "    \n",
    "    #加载ChatGLM3-6B模型\n",
    "    def load_model(self,\n",
    "                   model_name_or_path: str = \"chatglm3-6b\"):\n",
    "        self.tokenizer = AutoTokenizer.from_pretrained(\n",
    "            model_name_or_path,\n",
    "            trust_remote_code=True\n",
    "        )\n",
    "        self.model = AutoModel.from_pretrained(model_name_or_path, trust_remote_code=True).half().cuda()\n",
    "        self.model = self.model.eval()\n",
    "\n",
    "\n",
    "work_dir = '/home/ma-user/work'\n",
    "\n",
    "llm_model_name = 'chatglm3-6b'\n",
    "llm_model_path = os.path.join(work_dir, llm_model_name)\n",
    "\n",
    "chatLLM = ChatGLMService()\n",
    "chatLLM.load_model(model_name_or_path=llm_model_path)"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 3,
   "id": "5c0b4d09-1f16-4b4a-8476-7135decc508f",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "睡眠不好会导致头疼，可以尝试以下方法缓解头疼：\n",
      "\n",
      "1. 改善睡眠环境：保持卧室安静、舒适、黑暗。避免在睡前使用电子设备，因为蓝光可能会干扰你的睡眠。选择一款舒适的床垫和枕头，保证睡眠时身体得到充分支撑。\n",
      "\n",
      "2. 规律作息：尽量保持每天相同的入睡和起床时间，帮助身体建立稳定的生物钟。\n",
      "\n",
      "3. 放松身心：睡前进行一些放松活动，如深呼吸、瑜伽、阅读等，有助于减轻压力，促进入睡。\n",
      "\n",
      "4. 合理膳食：避免晚餐过量，尽量不要在临睡前摄入咖啡因、酒精等刺激性食物。\n",
      "\n",
      "5. 锻炼身体：适当的锻炼有助于改善睡眠质量，但避免在睡前进行剧烈运动。\n",
      "\n",
      "6. 头疼缓解方法：如果头疼持续不缓，可以尝试按摩头部、颈部，或用热毛巾敷在额头上。\n",
      "\n",
      "7. 避免过度用脑：白天避免过度思考问题，保持心情轻松，有助于晚上更好地入睡。\n",
      "\n",
      "如果头疼症状持续严重，建议及时就医，以便找出潜在的健康问题。\n"
     ]
    }
   ],
   "source": [
    "res = chatLLM('睡眠不好很头疼，有什么解决方案')\n",
    "print(res)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "e97d07ba-2c0a-4c3c-9556-c9f232f7d17f",
   "metadata": {},
   "source": [
    "### 3.2 提示模板定义\n",
    "\n",
    "提示模板的定义可以帮助模型更好的理解你的问题，以下为几个例子：\n",
    "\n",
    "#### 3.2.1 说明回答要求"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 5,
   "id": "5af7db3e-1f06-4c28-909f-3b39ff034168",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "请尽量简洁来回答用户的问题。\n",
      "问题:\n",
      "睡眠不好很头疼，有什么解决方案\n",
      "\n",
      "答案:\n",
      "改善睡眠环境、规律作息、放松身心、合理膳食、锻炼身体、缓解头疼方法。如有持续严重的头疼，建议就医。\n"
     ]
    }
   ],
   "source": [
    "from langchain import PromptTemplate\n",
    "\n",
    "def get_prompt_template1():\n",
    "    prompt_template1 = \"\"\"\\\n",
    "请尽量简洁来回答用户的问题。\n",
    "问题:\n",
    "{question}\n",
    "\"\"\"\n",
    "\n",
    "    prompt = PromptTemplate(template=prompt_template1,input_variables=[\"question\"])\n",
    "    return prompt\n",
    "                                             \n",
    "prompt1 = get_prompt_template1()\n",
    "\n",
    "question1 = '睡眠不好很头疼，有什么解决方案'\n",
    "message1 = prompt1.format(question=question1)\n",
    "print(message1)\n",
    "\n",
    "res1 = chatLLM(message1)\n",
    "print('答案:')\n",
    "print(res1)\n",
    "                                             "
   ]
  },
  {
   "cell_type": "markdown",
   "id": "0938f45a-9819-435f-82cb-b420f6696b3f",
   "metadata": {},
   "source": [
    "#### 3.2.2 基于已知信息回答"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 6,
   "id": "6b046d64-f654-4fa8-9214-6b917e5ea089",
   "metadata": {},
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "\n",
      "基于以下已知信息，简洁和专业的来回答用户的问题。\n",
      "如果无法从中得到答案，请说 \"根据已知信息无法回答该问题\" 或 \"没有提供足够的相关信息\"，不允许在答案中添加编造成分，答案请使用中文。\n",
      "已知内容:\n",
      "\n",
      "    晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:\n",
      "    1. 制定规律的睡眠时间表:保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯,使你更容易入睡。尽量在每天的相同时间上床,并在同一时间起床。\n",
      "    2. 创造一个舒适的睡眠环境:确保睡眠环境舒适,安静,黑暗且温度适宜。可以使用舒适的床上用品,并保持房间通风。\n",
      "    3. 放松身心:在睡前做些放松的活动,例如泡个热水澡,听些轻柔的音乐,阅读一些有趣的书籍等,有助于缓解紧张和焦虑,使你更容易入睡。\n",
      "    4. 避免饮用含有咖啡因的饮料:咖啡因是一种刺激性物质,会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料,例如咖啡,茶和可乐。\n",
      "    5. 避免在床上做与睡眠无关的事情:在床上做些与睡眠无关的事情,例如看电影,玩游戏或工作等,可能会干扰你的睡眠。\n",
      "    6. 尝试呼吸技巧:深呼吸是一种放松技巧,可以帮助你缓解紧张和焦虑,使你更容易入睡。试着慢慢吸气,保持几秒钟,然后缓慢呼气。\n",
      "    如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。\n",
      "\n",
      "问题:\n",
      "睡眠不好很头疼，有什么解决方案\n",
      "    \n",
      "答案:\n",
      "睡眠不好可能会导致头疼，以下是一些建议帮助缓解头疼并改善睡眠质量：\n",
      "\n",
      "1. 改善睡眠环境：保持规律的睡眠时间表，创造一个舒适、安静、黑暗且温度适宜的睡眠环境。\n",
      "2. 放松身心：睡前进行一些放松活动，如泡热水澡、听轻柔音乐、阅读有趣书籍等，以缓解紧张和焦虑。\n",
      "3. 避免刺激性饮料：避免在睡前饮用含有咖啡因的饮料，如咖啡、茶和可乐。\n",
      "4. 避免在床上做与睡眠无关的事情：在床上避免做与睡眠无关的事情，如看电影、玩游戏或工作等。\n",
      "5. 尝试呼吸技巧：深呼吸可以帮助你放松紧张和焦虑，从而更容易入睡。\n",
      "\n",
      "如果这些方法无法帮助你入睡，可以考虑咨询医生或睡眠专家，寻求进一步的建议。\n"
     ]
    }
   ],
   "source": [
    "from langchain import PromptTemplate\n",
    "\n",
    "def get_prompt_template2():\n",
    "    prompt_template2 = \"\"\"\n",
    "基于以下已知信息，简洁和专业的来回答用户的问题。\n",
    "如果无法从中得到答案，请说 \"根据已知信息无法回答该问题\" 或 \"没有提供足够的相关信息\"，不允许在答案中添加编造成分，答案请使用中文。\n",
    "已知内容:\n",
    "{context}\n",
    "问题:\n",
    "{question}\n",
    "    \"\"\"\n",
    "    prompt2 = PromptTemplate(template=prompt_template2,\n",
    "                            input_variables=[\"context\", \"question\"])\n",
    "    return prompt2\n",
    "\n",
    "prompt2 = get_prompt_template2()\n",
    "context2 = '''\n",
    "    晚上睡不着可能会让你感到焦虑或不舒服,但以下是一些可以帮助你入睡的方法:\n",
    "    1. 制定规律的睡眠时间表:保持规律的睡眠时间表可以帮助你建立健康的睡眠习惯,使你更容易入睡。尽量在每天的相同时间上床,并在同一时间起床。\n",
    "    2. 创造一个舒适的睡眠环境:确保睡眠环境舒适,安静,黑暗且温度适宜。可以使用舒适的床上用品,并保持房间通风。\n",
    "    3. 放松身心:在睡前做些放松的活动,例如泡个热水澡,听些轻柔的音乐,阅读一些有趣的书籍等,有助于缓解紧张和焦虑,使你更容易入睡。\n",
    "    4. 避免饮用含有咖啡因的饮料:咖啡因是一种刺激性物质,会影响你的睡眠质量。尽量避免在睡前饮用含有咖啡因的饮料,例如咖啡,茶和可乐。\n",
    "    5. 避免在床上做与睡眠无关的事情:在床上做些与睡眠无关的事情,例如看电影,玩游戏或工作等,可能会干扰你的睡眠。\n",
    "    6. 尝试呼吸技巧:深呼吸是一种放松技巧,可以帮助你缓解紧张和焦虑,使你更容易入睡。试着慢慢吸气,保持几秒钟,然后缓慢呼气。\n",
    "    如果这些方法无法帮助你入睡,你可以考虑咨询医生或睡眠专家,寻求进一步的建议。\n",
    "'''\n",
    "question2 = '睡眠不好很头疼，有什么解决方案'\n",
    "\n",
    "message2 = prompt2.format(context=context2, question=question2)\n",
    "print(message2)\n",
    "\n",
    "res2 = chatLLM(message2)\n",
    "print('答案:')\n",
    "print(res2)"
   ]
  },
  {
   "cell_type": "markdown",
   "id": "cd9403ac-cc07-448b-a18c-ae1ceb9c0550",
   "metadata": {},
   "source": [
    "### 作业1\n",
    "基于[课程2](https://developer.huaweicloud.com/develop/aigallery/notebook/detail?id=2ebc801b-d828-42af-a6ec-5b3893993a7c)和课程3（本案例），实现一个基于ChatGLM3+LangChain的聊天应用，需要有Gradio界面。"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 1,
   "id": "9b1aead6-c9da-4346-b51d-41d8db61c1c2",
   "metadata": {
    "scrolled": true,
    "tags": []
   },
   "outputs": [
    {
     "name": "stdout",
     "output_type": "stream",
     "text": [
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Requirement already satisfied: transformers==4.30.2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (4.30.2)\n",
      "Requirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (3.12.2)\n",
      "Requirement already satisfied: huggingface-hub<1.0,>=0.14.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (0.20.1)\n",
      "Requirement already satisfied: numpy>=1.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (1.25.2)\n",
      "Requirement already satisfied: packaging>=20.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (23.2)\n",
      "Requirement already satisfied: pyyaml>=5.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (6.0.1)\n",
      "Requirement already satisfied: regex!=2019.12.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (2023.5.5)\n",
      "Requirement already satisfied: requests in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (2.31.0)\n",
      "Requirement already satisfied: tokenizers!=0.11.3,<0.14,>=0.11.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (0.13.3)\n",
      "Requirement already satisfied: safetensors>=0.3.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (0.3.3)\n",
      "Requirement already satisfied: tqdm>=4.27 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from transformers==4.30.2) (4.66.1)\n",
      "Requirement already satisfied: fsspec>=2023.5.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2) (2023.12.2)\n",
      "Requirement already satisfied: typing-extensions>=3.7.4.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub<1.0,>=0.14.1->transformers==4.30.2) (4.8.0)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->transformers==4.30.2) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->transformers==4.30.2) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->transformers==4.30.2) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests->transformers==4.30.2) (2023.11.17)\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Requirement already satisfied: sentencepiece==0.1.99 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (0.1.99)\n",
      "Looking in indexes: http://repo.myhuaweicloud.com/repository/pypi/simple\n",
      "Requirement already satisfied: torch==2.0.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (2.0.1)\n",
      "Requirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (3.12.2)\n",
      "Requirement already satisfied: typing-extensions in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (4.8.0)\n",
      "Requirement already satisfied: sympy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (1.12)\n",
      "Requirement already satisfied: networkx in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (2.8.8)\n",
      "Requirement already satisfied: jinja2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (3.1.2)\n",
      "Requirement already satisfied: nvidia-cuda-nvrtc-cu11==11.7.99 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-runtime-cu11==11.7.99 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (11.7.99)\n",
      "Requirement already satisfied: nvidia-cuda-cupti-cu11==11.7.101 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (11.7.101)\n",
      "Requirement already satisfied: nvidia-cudnn-cu11==8.5.0.96 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (8.5.0.96)\n",
      "Requirement already satisfied: nvidia-cublas-cu11==11.10.3.66 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (11.10.3.66)\n",
      "Requirement already satisfied: nvidia-cufft-cu11==10.9.0.58 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (10.9.0.58)\n",
      "Requirement already satisfied: nvidia-curand-cu11==10.2.10.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (10.2.10.91)\n",
      "Requirement already satisfied: nvidia-cusolver-cu11==11.4.0.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (11.4.0.1)\n",
      "Requirement already satisfied: nvidia-cusparse-cu11==11.7.4.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (11.7.4.91)\n",
      "Requirement already satisfied: nvidia-nccl-cu11==2.14.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (2.14.3)\n",
      "Requirement already satisfied: nvidia-nvtx-cu11==11.7.91 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (11.7.91)\n",
      "Requirement already satisfied: triton==2.0.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from torch==2.0.1) (2.0.0)\n",
      "Requirement already satisfied: setuptools in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (68.2.2)\n",
      "Requirement already satisfied: wheel in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from nvidia-cublas-cu11==11.10.3.66->torch==2.0.1) (0.41.2)\n",
      "Requirement already satisfied: cmake in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from triton==2.0.0->torch==2.0.1) (3.27.2)\n",
      "Requirement already satisfied: lit in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from triton==2.0.0->torch==2.0.1) (16.0.5)\n",
      "Requirement already satisfied: MarkupSafe>=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from jinja2->torch==2.0.1) (2.1.3)\n",
      "Requirement already satisfied: mpmath>=0.19 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from sympy->torch==2.0.1) (1.3.0)\n",
      "Looking in indexes: https://pypi.tuna.tsinghua.edu.cn/simple\n",
      "Requirement already satisfied: gradio==3.39 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (3.39.0)\n",
      "Requirement already satisfied: mdtex2html in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (1.2.0)\n",
      "Requirement already satisfied: aiofiles<24.0,>=22.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (23.2.1)\n",
      "Requirement already satisfied: aiohttp~=3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (3.9.1)\n",
      "Requirement already satisfied: altair<6.0,>=4.2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (5.2.0)\n",
      "Requirement already satisfied: fastapi in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.106.0)\n",
      "Requirement already satisfied: ffmpy in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.3.1)\n",
      "Requirement already satisfied: gradio-client>=0.3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.8.0)\n",
      "Requirement already satisfied: httpx in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.26.0)\n",
      "Requirement already satisfied: huggingface-hub>=0.14.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.20.1)\n",
      "Requirement already satisfied: jinja2<4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (3.1.2)\n",
      "Requirement already satisfied: markdown-it-py>=2.0.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.39) (2.2.0)\n",
      "Requirement already satisfied: markupsafe~=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.1.3)\n",
      "Requirement already satisfied: matplotlib~=3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (3.8.2)\n",
      "Requirement already satisfied: mdit-py-plugins<=0.3.3 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.3.3)\n",
      "Requirement already satisfied: numpy~=1.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (1.25.2)\n",
      "Requirement already satisfied: orjson~=3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (3.9.10)\n",
      "Requirement already satisfied: packaging in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (23.2)\n",
      "Requirement already satisfied: pandas<3.0,>=1.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.1.4)\n",
      "Requirement already satisfied: pillow<11.0,>=8.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (10.1.0)\n",
      "Requirement already satisfied: pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.5.3)\n",
      "Requirement already satisfied: pydub in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.25.1)\n",
      "Requirement already satisfied: python-multipart in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.0.6)\n",
      "Requirement already satisfied: pyyaml<7.0,>=5.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (6.0.1)\n",
      "Requirement already satisfied: requests~=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.31.0)\n",
      "Requirement already satisfied: semantic-version~=2.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (2.10.0)\n",
      "Requirement already satisfied: typing-extensions~=4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (4.8.0)\n",
      "Requirement already satisfied: uvicorn>=0.14.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (0.25.0)\n",
      "Requirement already satisfied: websockets<12.0,>=10.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio==3.39) (11.0.3)\n",
      "Requirement already satisfied: markdown in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from mdtex2html) (3.5.1)\n",
      "Requirement already satisfied: latex2mathml in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from mdtex2html) (3.77.0)\n",
      "Requirement already satisfied: attrs>=17.3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (23.1.0)\n",
      "Requirement already satisfied: multidict<7.0,>=4.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (6.0.4)\n",
      "Requirement already satisfied: yarl<2.0,>=1.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (1.9.4)\n",
      "Requirement already satisfied: frozenlist>=1.1.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (1.4.1)\n",
      "Requirement already satisfied: aiosignal>=1.1.2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (1.3.1)\n",
      "Requirement already satisfied: async-timeout<5.0,>=4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from aiohttp~=3.0->gradio==3.39) (4.0.3)\n",
      "Requirement already satisfied: jsonschema>=3.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from altair<6.0,>=4.2.0->gradio==3.39) (4.20.0)\n",
      "Requirement already satisfied: toolz in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from altair<6.0,>=4.2.0->gradio==3.39) (0.12.0)\n",
      "Requirement already satisfied: fsspec in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from gradio-client>=0.3.0->gradio==3.39) (2023.12.2)\n",
      "Requirement already satisfied: filelock in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.14.0->gradio==3.39) (3.12.2)\n",
      "Requirement already satisfied: tqdm>=4.42.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from huggingface-hub>=0.14.0->gradio==3.39) (4.66.1)\n",
      "Requirement already satisfied: mdurl~=0.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from markdown-it-py>=2.0.0->markdown-it-py[linkify]>=2.0.0->gradio==3.39) (0.1.2)\n",
      "Requirement already satisfied: linkify-it-py<3,>=1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from markdown-it-py[linkify]>=2.0.0->gradio==3.39) (2.0.2)\n",
      "Requirement already satisfied: contourpy>=1.0.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.39) (1.2.0)\n",
      "Requirement already satisfied: cycler>=0.10 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.39) (0.12.1)\n",
      "Requirement already satisfied: fonttools>=4.22.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.39) (4.47.0)\n",
      "Requirement already satisfied: kiwisolver>=1.3.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.39) (1.4.5)\n",
      "Requirement already satisfied: pyparsing>=2.3.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.39) (3.1.1)\n",
      "Requirement already satisfied: python-dateutil>=2.7 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from matplotlib~=3.0->gradio==3.39) (2.8.2)\n",
      "Requirement already satisfied: pytz>=2020.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pandas<3.0,>=1.0->gradio==3.39) (2023.3.post1)\n",
      "Requirement already satisfied: tzdata>=2022.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pandas<3.0,>=1.0->gradio==3.39) (2023.3)\n",
      "Requirement already satisfied: annotated-types>=0.4.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio==3.39) (0.6.0)\n",
      "Requirement already satisfied: pydantic-core==2.14.6 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from pydantic!=1.8,!=1.8.1,!=2.0.0,!=2.0.1,<3.0.0,>=1.7.4->gradio==3.39) (2.14.6)\n",
      "Requirement already satisfied: charset-normalizer<4,>=2 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests~=2.0->gradio==3.39) (3.3.2)\n",
      "Requirement already satisfied: idna<4,>=2.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests~=2.0->gradio==3.39) (3.4)\n",
      "Requirement already satisfied: urllib3<3,>=1.21.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests~=2.0->gradio==3.39) (2.1.0)\n",
      "Requirement already satisfied: certifi>=2017.4.17 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from requests~=2.0->gradio==3.39) (2023.11.17)\n",
      "Requirement already satisfied: click>=7.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from uvicorn>=0.14.0->gradio==3.39) (8.1.7)\n",
      "Requirement already satisfied: h11>=0.8 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from uvicorn>=0.14.0->gradio==3.39) (0.14.0)\n",
      "Requirement already satisfied: anyio<4.0.0,>=3.7.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from fastapi->gradio==3.39) (3.7.1)\n",
      "Requirement already satisfied: starlette<0.28.0,>=0.27.0 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from fastapi->gradio==3.39) (0.27.0)\n",
      "Requirement already satisfied: httpcore==1.* in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from httpx->gradio==3.39) (1.0.2)\n",
      "Requirement already satisfied: sniffio in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from httpx->gradio==3.39) (1.3.0)\n",
      "Requirement already satisfied: exceptiongroup in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from anyio<4.0.0,>=3.7.1->fastapi->gradio==3.39) (1.2.0)\n",
      "Requirement already satisfied: jsonschema-specifications>=2023.03.6 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.39) (2023.12.1)\n",
      "Requirement already satisfied: referencing>=0.28.4 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.39) (0.32.0)\n",
      "Requirement already satisfied: rpds-py>=0.7.1 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from jsonschema>=3.0->altair<6.0,>=4.2.0->gradio==3.39) (0.15.2)\n",
      "Requirement already satisfied: uc-micro-py in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from linkify-it-py<3,>=1->markdown-it-py[linkify]>=2.0.0->gradio==3.39) (1.0.2)\n",
      "Requirement already satisfied: six>=1.5 in /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages (from python-dateutil>=2.7->matplotlib~=3.0->gradio==3.39) (1.16.0)\n"
     ]
    }
   ],
   "source": [
    "!pip install transformers==4.30.2\n",
    "!pip install sentencepiece==0.1.99\n",
    "!pip install torch==2.0.1\n",
    "!pip install gradio==3.39 mdtex2html -i https://pypi.tuna.tsinghua.edu.cn/simple --trusted-host pypi.tuna.tsinghua.edu.cn\n",
    "!cp /home/ma-user/work/frpc_linux_amd64 /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2\n",
    "!chmod +x /home/ma-user/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/gradio/frpc_linux_amd64_v0.2"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": 4,
   "id": "0b6fff57-d324-4d54-b529-bc2c293689bb",
   "metadata": {},
   "outputs": [
    {
     "name": "stderr",
     "output_type": "stream",
     "text": [
      "Loading checkpoint shards: 100%|██████████| 7/7 [00:08<00:00,  1.25s/it]\n"
     ]
    },
    {
     "ename": "OutOfMemoryError",
     "evalue": "CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 15.90 GiB total capacity; 15.21 GiB already allocated; 11.75 MiB free; 15.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF",
     "output_type": "error",
     "traceback": [
      "\u001b[0;31m---------------------------------------------------------------------------\u001b[0m",
      "\u001b[0;31mOutOfMemoryError\u001b[0m                          Traceback (most recent call last)",
      "Cell \u001b[0;32mIn[4], line 50\u001b[0m\n\u001b[1;32m     48\u001b[0m llm_model_path \u001b[38;5;241m=\u001b[39m os\u001b[38;5;241m.\u001b[39mpath\u001b[38;5;241m.\u001b[39mjoin(work_dir, llm_model_name)\n\u001b[1;32m     49\u001b[0m chatLLM \u001b[38;5;241m=\u001b[39m ChatGLMService()\n\u001b[0;32m---> 50\u001b[0m \u001b[43mchatLLM\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mload_model\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel_name_or_path\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[43mllm_model_path\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     52\u001b[0m \u001b[38;5;66;03m#去除输入框的内容\u001b[39;00m\n\u001b[1;32m     53\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mreset_user_input\u001b[39m():\n",
      "Cell \u001b[0;32mIn[4], line 42\u001b[0m, in \u001b[0;36mChatGLMService.load_model\u001b[0;34m(self, model_name_or_path)\u001b[0m\n\u001b[1;32m     37\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mload_model\u001b[39m(\u001b[38;5;28mself\u001b[39m, model_name_or_path: \u001b[38;5;28mstr\u001b[39m \u001b[38;5;241m=\u001b[39m \u001b[38;5;124m\"\u001b[39m\u001b[38;5;124mchatglm3-6b\u001b[39m\u001b[38;5;124m\"\u001b[39m):\n\u001b[1;32m     38\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mtokenizer \u001b[38;5;241m=\u001b[39m AutoTokenizer\u001b[38;5;241m.\u001b[39mfrom_pretrained(\n\u001b[1;32m     39\u001b[0m         model_name_or_path,\n\u001b[1;32m     40\u001b[0m         trust_remote_code\u001b[38;5;241m=\u001b[39m\u001b[38;5;28;01mTrue\u001b[39;00m\n\u001b[1;32m     41\u001b[0m     )\n\u001b[0;32m---> 42\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel \u001b[38;5;241m=\u001b[39m \u001b[43mAutoModel\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mfrom_pretrained\u001b[49m\u001b[43m(\u001b[49m\u001b[43mmodel_name_or_path\u001b[49m\u001b[43m,\u001b[49m\u001b[43m \u001b[49m\u001b[43mtrust_remote_code\u001b[49m\u001b[38;5;241;43m=\u001b[39;49m\u001b[38;5;28;43;01mTrue\u001b[39;49;00m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mhalf\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcuda\u001b[49m\u001b[43m(\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m     43\u001b[0m     \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel \u001b[38;5;241m=\u001b[39m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mmodel\u001b[38;5;241m.\u001b[39meval()\n",
      "File \u001b[0;32m~/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/nn/modules/module.py:905\u001b[0m, in \u001b[0;36mModule.cuda\u001b[0;34m(self, device)\u001b[0m\n\u001b[1;32m    888\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcuda\u001b[39m(\u001b[38;5;28mself\u001b[39m: T, device: Optional[Union[\u001b[38;5;28mint\u001b[39m, device]] \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m T:\n\u001b[1;32m    889\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124;03m\"\"\"Moves all model parameters and buffers to the GPU.\u001b[39;00m\n\u001b[1;32m    890\u001b[0m \n\u001b[1;32m    891\u001b[0m \u001b[38;5;124;03m    This also makes associated parameters and buffers different objects. So\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    903\u001b[0m \u001b[38;5;124;03m        Module: self\u001b[39;00m\n\u001b[1;32m    904\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 905\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28;43mself\u001b[39;49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_apply\u001b[49m\u001b[43m(\u001b[49m\u001b[38;5;28;43;01mlambda\u001b[39;49;00m\u001b[43m \u001b[49m\u001b[43mt\u001b[49m\u001b[43m:\u001b[49m\u001b[43m \u001b[49m\u001b[43mt\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcuda\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdevice\u001b[49m\u001b[43m)\u001b[49m\u001b[43m)\u001b[49m\n",
      "File \u001b[0;32m~/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/nn/modules/module.py:797\u001b[0m, in \u001b[0;36mModule._apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    795\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_apply\u001b[39m(\u001b[38;5;28mself\u001b[39m, fn):\n\u001b[1;32m    796\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m module \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mchildren():\n\u001b[0;32m--> 797\u001b[0m         \u001b[43mmodule\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_apply\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfn\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    799\u001b[0m     \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcompute_should_use_set_data\u001b[39m(tensor, tensor_applied):\n\u001b[1;32m    800\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m torch\u001b[38;5;241m.\u001b[39m_has_compatible_shallow_copy_type(tensor, tensor_applied):\n\u001b[1;32m    801\u001b[0m             \u001b[38;5;66;03m# If the new tensor has compatible tensor type as the existing tensor,\u001b[39;00m\n\u001b[1;32m    802\u001b[0m             \u001b[38;5;66;03m# the current behavior is to change the tensor in-place using `.data =`,\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    807\u001b[0m             \u001b[38;5;66;03m# global flag to let the user control whether they want the future\u001b[39;00m\n\u001b[1;32m    808\u001b[0m             \u001b[38;5;66;03m# behavior of overwriting the existing tensor or not.\u001b[39;00m\n",
      "File \u001b[0;32m~/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/nn/modules/module.py:797\u001b[0m, in \u001b[0;36mModule._apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    795\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_apply\u001b[39m(\u001b[38;5;28mself\u001b[39m, fn):\n\u001b[1;32m    796\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m module \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mchildren():\n\u001b[0;32m--> 797\u001b[0m         \u001b[43mmodule\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_apply\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfn\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    799\u001b[0m     \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcompute_should_use_set_data\u001b[39m(tensor, tensor_applied):\n\u001b[1;32m    800\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m torch\u001b[38;5;241m.\u001b[39m_has_compatible_shallow_copy_type(tensor, tensor_applied):\n\u001b[1;32m    801\u001b[0m             \u001b[38;5;66;03m# If the new tensor has compatible tensor type as the existing tensor,\u001b[39;00m\n\u001b[1;32m    802\u001b[0m             \u001b[38;5;66;03m# the current behavior is to change the tensor in-place using `.data =`,\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    807\u001b[0m             \u001b[38;5;66;03m# global flag to let the user control whether they want the future\u001b[39;00m\n\u001b[1;32m    808\u001b[0m             \u001b[38;5;66;03m# behavior of overwriting the existing tensor or not.\u001b[39;00m\n",
      "    \u001b[0;31m[... skipping similar frames: Module._apply at line 797 (3 times)]\u001b[0m\n",
      "File \u001b[0;32m~/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/nn/modules/module.py:797\u001b[0m, in \u001b[0;36mModule._apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    795\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21m_apply\u001b[39m(\u001b[38;5;28mself\u001b[39m, fn):\n\u001b[1;32m    796\u001b[0m     \u001b[38;5;28;01mfor\u001b[39;00m module \u001b[38;5;129;01min\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39mchildren():\n\u001b[0;32m--> 797\u001b[0m         \u001b[43mmodule\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43m_apply\u001b[49m\u001b[43m(\u001b[49m\u001b[43mfn\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    799\u001b[0m     \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcompute_should_use_set_data\u001b[39m(tensor, tensor_applied):\n\u001b[1;32m    800\u001b[0m         \u001b[38;5;28;01mif\u001b[39;00m torch\u001b[38;5;241m.\u001b[39m_has_compatible_shallow_copy_type(tensor, tensor_applied):\n\u001b[1;32m    801\u001b[0m             \u001b[38;5;66;03m# If the new tensor has compatible tensor type as the existing tensor,\u001b[39;00m\n\u001b[1;32m    802\u001b[0m             \u001b[38;5;66;03m# the current behavior is to change the tensor in-place using `.data =`,\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    807\u001b[0m             \u001b[38;5;66;03m# global flag to let the user control whether they want the future\u001b[39;00m\n\u001b[1;32m    808\u001b[0m             \u001b[38;5;66;03m# behavior of overwriting the existing tensor or not.\u001b[39;00m\n",
      "File \u001b[0;32m~/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/nn/modules/module.py:820\u001b[0m, in \u001b[0;36mModule._apply\u001b[0;34m(self, fn)\u001b[0m\n\u001b[1;32m    816\u001b[0m \u001b[38;5;66;03m# Tensors stored in modules are graph leaves, and we don't want to\u001b[39;00m\n\u001b[1;32m    817\u001b[0m \u001b[38;5;66;03m# track autograd history of `param_applied`, so we have to use\u001b[39;00m\n\u001b[1;32m    818\u001b[0m \u001b[38;5;66;03m# `with torch.no_grad():`\u001b[39;00m\n\u001b[1;32m    819\u001b[0m \u001b[38;5;28;01mwith\u001b[39;00m torch\u001b[38;5;241m.\u001b[39mno_grad():\n\u001b[0;32m--> 820\u001b[0m     param_applied \u001b[38;5;241m=\u001b[39m \u001b[43mfn\u001b[49m\u001b[43m(\u001b[49m\u001b[43mparam\u001b[49m\u001b[43m)\u001b[49m\n\u001b[1;32m    821\u001b[0m should_use_set_data \u001b[38;5;241m=\u001b[39m compute_should_use_set_data(param, param_applied)\n\u001b[1;32m    822\u001b[0m \u001b[38;5;28;01mif\u001b[39;00m should_use_set_data:\n",
      "File \u001b[0;32m~/anaconda3/envs/python-3.10.10/lib/python3.10/site-packages/torch/nn/modules/module.py:905\u001b[0m, in \u001b[0;36mModule.cuda.<locals>.<lambda>\u001b[0;34m(t)\u001b[0m\n\u001b[1;32m    888\u001b[0m \u001b[38;5;28;01mdef\u001b[39;00m \u001b[38;5;21mcuda\u001b[39m(\u001b[38;5;28mself\u001b[39m: T, device: Optional[Union[\u001b[38;5;28mint\u001b[39m, device]] \u001b[38;5;241m=\u001b[39m \u001b[38;5;28;01mNone\u001b[39;00m) \u001b[38;5;241m-\u001b[39m\u001b[38;5;241m>\u001b[39m T:\n\u001b[1;32m    889\u001b[0m \u001b[38;5;250m    \u001b[39m\u001b[38;5;124mr\u001b[39m\u001b[38;5;124;03m\"\"\"Moves all model parameters and buffers to the GPU.\u001b[39;00m\n\u001b[1;32m    890\u001b[0m \n\u001b[1;32m    891\u001b[0m \u001b[38;5;124;03m    This also makes associated parameters and buffers different objects. So\u001b[39;00m\n\u001b[0;32m   (...)\u001b[0m\n\u001b[1;32m    903\u001b[0m \u001b[38;5;124;03m        Module: self\u001b[39;00m\n\u001b[1;32m    904\u001b[0m \u001b[38;5;124;03m    \"\"\"\u001b[39;00m\n\u001b[0;32m--> 905\u001b[0m     \u001b[38;5;28;01mreturn\u001b[39;00m \u001b[38;5;28mself\u001b[39m\u001b[38;5;241m.\u001b[39m_apply(\u001b[38;5;28;01mlambda\u001b[39;00m t: \u001b[43mt\u001b[49m\u001b[38;5;241;43m.\u001b[39;49m\u001b[43mcuda\u001b[49m\u001b[43m(\u001b[49m\u001b[43mdevice\u001b[49m\u001b[43m)\u001b[49m)\n",
      "\u001b[0;31mOutOfMemoryError\u001b[0m: CUDA out of memory. Tried to allocate 36.00 MiB (GPU 0; 15.90 GiB total capacity; 15.21 GiB already allocated; 11.75 MiB free; 15.22 GiB reserved in total by PyTorch) If reserved memory is >> allocated memory try setting max_split_size_mb to avoid fragmentation.  See documentation for Memory Management and PYTORCH_CUDA_ALLOC_CONF"
     ]
    }
   ],
   "source": [
    "import os\n",
    "import gradio as gr\n",
    "from langchain.llms.base import LLM\n",
    "from langchain.llms.utils import enforce_stop_tokens\n",
    "from transformers import AutoModel, AutoTokenizer\n",
    "\n",
    "# 新建 ChatGLMService 类\n",
    "class ChatGLMService(LLM):\n",
    "    max_token: int = 10000\n",
    "    temperature: float = 0.1\n",
    "    top_p = 0.9\n",
    "    history = []\n",
    "    tokenizer: object = None\n",
    "    model: object = None\n",
    "\n",
    "    def __init__(self):\n",
    "        super().__init__()\n",
    "\n",
    "    @property\n",
    "    def _llm_type(self) -> str:\n",
    "        return \"ChatGLM\"\n",
    "\n",
    "    def _call(self, prompt: str, stop=None) -> str:\n",
    "        response, history = self.model.chat(\n",
    "            self.tokenizer,\n",
    "            prompt,\n",
    "            history=self.history,\n",
    "            max_length=self.max_token,\n",
    "            temperature=self.temperature,\n",
    "        )\n",
    "        if stop is not None:\n",
    "            response = enforce_stop_tokens(response, stop)\n",
    "\n",
    "        self.history = self.history + [history[-1]]\n",
    "        return response\n",
    "\n",
    "    def load_model(self, model_name_or_path: str = \"chatglm3-6b\"):\n",
    "        self.tokenizer = AutoTokenizer.from_pretrained(\n",
    "            model_name_or_path,\n",
    "            trust_remote_code=True\n",
    "        )\n",
    "        self.model = AutoModel.from_pretrained(model_name_or_path, trust_remote_code=True).half().cuda()\n",
    "        self.model = self.model.eval()\n",
    "\n",
    "# 创建 ChatGLMService 实例并加载模型\n",
    "work_dir = '/home/ma-user/work'\n",
    "llm_model_name = 'chatglm3-6b'\n",
    "llm_model_path = os.path.join(work_dir, llm_model_name)\n",
    "chatLLM = ChatGLMService()\n",
    "chatLLM.load_model(model_name_or_path=llm_model_path)\n",
    "\n",
    "#去除输入框的内容\n",
    "def reset_user_input():\n",
    "    return gr.update(value='')\n",
    "\n",
    "#清除状态\n",
    "def reset_state():\n",
    "    return [], [], None\n",
    "\n",
    "\n",
    "\n",
    "# Gradio 界面\n",
    "with gr.Blocks() as demo:\n",
    "    gr.HTML(\"\"\"<h1 align=\"center\">ChatGLM3-6B</h1>\"\"\")\n",
    "\n",
    "    chatbot = gr.Chatbot()\n",
    "    with gr.Row():\n",
    "        with gr.Column(scale=4):\n",
    "            with gr.Column(scale=12):\n",
    "                user_input = gr.Textbox(show_label=False, placeholder=\"Input...\", lines=10, container=False)\n",
    "            with gr.Column(min_width=32, scale=1):\n",
    "                submitBtn = gr.Button(\"Submit\", variant=\"primary\")\n",
    "        with gr.Column(scale=1):\n",
    "            emptyBtn = gr.Button(\"Clear History\")\n",
    "            max_length = gr.Slider(0, 32768, value=8192, step=1.0, label=\"Maximum length\", interactive=True)\n",
    "            top_p = gr.Slider(0, 1, value=0.8, step=0.01, label=\"Top P\", interactive=True)\n",
    "            temperature = gr.Slider(0, 1, value=0.95, step=0.01, label=\"Temperature\", interactive=True)\n",
    "\n",
    "    history = gr.State([])\n",
    "    past_key_values = gr.State(None)\n",
    "\n",
    "    def predict(input, chatbot, max_length, top_p, temperature, history, past_key_values):\n",
    "        chatbot.append((input, chatLLM(input, stop=[\"<eos>\"])))\n",
    "        for response, history, past_key_values in chatLLM.stream_chat(\n",
    "            input, history, past_key_values=past_key_values,\n",
    "            return_past_key_values=True,\n",
    "            max_length=int(max_length), top_p=float(top_p),\n",
    "            temperature=float(temperature)\n",
    "        ):\n",
    "            chatbot[-1] = (input, response)\n",
    "            yield chatbot, history, past_key_values\n",
    "\n",
    "    submitBtn.click(predict, [user_input, chatbot, max_length, top_p, temperature, history, past_key_values],\n",
    "                    [chatbot, history, past_key_values], show_progress=True)\n",
    "    submitBtn.click(reset_user_input, [], [user_input])\n",
    "\n",
    "    emptyBtn.click(reset_state, outputs=[chatbot, history, past_key_values], show_progress=True)\n",
    "\n",
    "demo.queue().launch(share=True, inbrowser=True)\n"
   ]
  },
  {
   "cell_type": "code",
   "execution_count": null,
   "id": "a05993fa-7660-4488-bf78-b2fc39a013ce",
   "metadata": {},
   "outputs": [],
   "source": []
  }
 ],
 "metadata": {
  "AIGalleryInfo": {
   "item_id": "4bcbac7c-04a9-4331-8191-19382ac406a8"
  },
  "flavorInfo": {
   "architecture": "X86_64",
   "category": "GPU"
  },
  "imageInfo": {
   "id": "d996b661-e127-48c4-a90a-fca29535f201",
   "name": "pytorch1.10-cuda10.2-cudnn7-ubuntu18.04"
  },
  "kernelspec": {
   "display_name": "python-3.7.10",
   "language": "python",
   "name": "python-3.7.10"
  },
  "language_info": {
   "codemirror_mode": {
    "name": "ipython",
    "version": 3
   },
   "file_extension": ".py",
   "mimetype": "text/x-python",
   "name": "python",
   "nbconvert_exporter": "python",
   "pygments_lexer": "ipython3",
   "version": "3.7.10"
  }
 },
 "nbformat": 4,
 "nbformat_minor": 5
}
